{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TensorGraph Layers and TensorFlow eager"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " In this tutorial we will look at the working of TensorGraph layer with TensorFlow eager.\n",
    " But before that let's see what exactly is TensorFlow eager."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eager execution is an imperative, define-by-run interface where operations are executed immediately as they are called from Python. In other words, eager execution is a feature that makes TensorFlow execute operations immediately. Concrete values are returned instead of a computational graph to be executed later.\n",
    "As a result:\n",
    "- It allows writing imperative coding style like numpy\n",
    "- Provides fast debugging with immediate run-time errors and integration with Python tools\n",
    "- Strong support for higher-order gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/nitin/anaconda3/envs/deepchem/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow.contrib.eager as tfe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After importing neccessary modules, at the program startup we invoke `enable_eager_execution()`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfe.enable_eager_execution()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enabling eager execution changes how TensorFlow functions behave. Tensor objects return concrete values instead of being a symbolic reference to nodes in a static computational graph(non-eager mode). As a result, eager execution should be enabled at the beginning of a program."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that with eager execution enabled, these operations consume and return multi-dimensional arrays as `Tensor` objects, similar to NumPy `ndarrays`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dense layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import deepchem as dc\n",
    "from deepchem.models.tensorgraph import layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following snippet we describe how to create a `Dense` layer in eager mode. The good thing about calling a layer as a function is that we don't have to call `create_tensor()` directly. This is identical to tensorflow API and has no conflict. And since eager mode is enabled, it should return concrete tensors right away."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[-0.01966965 -0.36708647 -1.2114673 ]\n",
      " [ 0.4068503  -0.1943181  -1.1845192 ]\n",
      " [-0.1708395  -0.4865471  -1.4172423 ]\n",
      " [ 0.18667166 -0.24077846 -1.0544432 ]\n",
      " [ 0.47903097 -0.5307548  -2.4122753 ]\n",
      " [ 0.215595   -0.3206781  -1.3613585 ]\n",
      " [ 0.53623885 -0.29272282 -1.6845896 ]\n",
      " [ 0.33747557 -0.49973956 -2.1234663 ]\n",
      " [ 0.6471517  -0.29422784 -1.8340569 ]\n",
      " [ 0.52365005 -0.25159127 -1.5295881 ]], shape=(10, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# Initialize parameters\n",
    "in_dim = 2\n",
    "out_dim = 3\n",
    "batch_size = 10\n",
    "\n",
    "inputs = np.random.rand(batch_size, in_dim).astype(np.float32) #Input \n",
    "\n",
    "layer = layers.Dense(out_dim) # Provide the number of output values as parameter. This creates a Dense layer\n",
    "result = layer(inputs) #get the ouput tensors\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating a second `Dense` layer should produce different results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[-0.2995599  -0.94036776 -0.12444651]\n",
      " [-0.7348614  -0.3941293  -0.6263947 ]\n",
      " [-0.19710277 -1.2823544   0.02952629]\n",
      " [-0.47212085 -0.5672255  -0.34971827]\n",
      " [-1.1339976  -1.2335718  -0.86162925]\n",
      " [-0.5831823  -0.7636563  -0.42140976]\n",
      " [-1.0011474  -0.6127606  -0.84064883]\n",
      " [-0.9108876  -1.189698   -0.6587274 ]\n",
      " [-1.1556706  -0.5890429  -0.9902593 ]\n",
      " [-0.9471516  -0.5110685  -0.80683327]], shape=(10, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "layer2 = layers.Dense(out_dim)\n",
    "result2 = layer2(inputs)\n",
    "\n",
    "print(result2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also execute the layer in eager mode to compute its output as a function of inputs. If the layer defines any variables, they are created the first time it is invoked. This happens in the same exact way that we would create a single layer in non-eager mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following is also a way to create a layer in eager mode. The `create_tensor()` is invoked by `__call__()` object. This gives us an advantage of directly passing the tensor as a parameter while constructing a TensorGraph layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 0.5611591  -0.22970255  0.11273235]\n",
      " [ 0.07900402 -0.62115765 -0.75677997]\n",
      " [ 0.8194277  -0.13113126  0.43268698]\n",
      " [ 0.263784   -0.38960344 -0.31656086]\n",
      " [ 0.5461697  -0.939682   -0.82996976]\n",
      " [ 0.36857378 -0.4793542  -0.35699493]\n",
      " [ 0.15906434 -0.84395605 -0.990051  ]\n",
      " [ 0.57359767 -0.7488086  -0.559262  ]\n",
      " [ 0.10336368 -0.9777839  -1.2067692 ]\n",
      " [ 0.10391745 -0.8005077  -0.9737376 ]], shape=(10, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "x = layers.Dense(out_dim)(inputs)\n",
    "\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conv1D layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Dense` layers are one of the layers defined in Deepchem. Along with it there are several others like `Conv1D`, `Conv2D`, `conv3D` etc. We also take a look at how to construct a `Conv1D` layer below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically this layer creates a convolution kernel that is convolved with the layer input over a single spatial (or temporal) dimension to produce a tensor of outputs.\n",
    "When using this layer as the first layer in a model, provide an `input_shape` argument (tuple of integers or `None`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the argument `input_shape` is passed in as a tuple of integers e.g (2, 3) it would mean we are passing a sequence of 2 vectors of 3-Dimensional vectors.\n",
    "And when it is passed as (None, 3) it means that we want variable-length sequences of 3-dimensional vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepchem.models.tensorgraph.layers import Conv1D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[-0.4163494   0.82620853 -0.19792846]\n",
      "  [ 0.11079708  0.63339764 -0.4348114 ]\n",
      "  [-0.08490932  0.9501296  -0.78048754]\n",
      "  [-0.11654931  1.3595564  -0.81889915]]\n",
      "\n",
      " [[-0.04658069  0.32090923 -0.01684245]\n",
      "  [ 0.42333373  0.5015999  -0.51524043]\n",
      "  [-0.14566499  1.3786271  -0.8008883 ]\n",
      "  [-0.39936984  1.0127177  -0.41432446]]\n",
      "\n",
      " [[-0.36023882  1.2730068  -0.6852635 ]\n",
      "  [-0.33337924  0.9590143  -0.4819267 ]\n",
      "  [-0.17271379  0.38323316 -0.16304907]\n",
      "  [ 0.03019462  0.4663608  -0.29353675]]\n",
      "\n",
      " [[-0.37523568  1.1785955  -0.6067249 ]\n",
      "  [-0.3328942   1.0252076  -0.6913832 ]\n",
      "  [-0.32247764  0.54115754 -0.28387934]\n",
      "  [ 0.07658018  0.7736342  -0.5747899 ]]\n",
      "\n",
      " [[-0.51029617  0.92260885 -0.25738803]\n",
      "  [ 0.00583639  0.16965131 -0.01829494]\n",
      "  [ 0.21363875  0.17591321 -0.17815764]\n",
      "  [ 0.22821559  0.739758   -0.5263671 ]]], shape=(5, 4, 3), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "width = 5\n",
    "in_channels = 2\n",
    "filters = 3\n",
    "kernel_size = 2\n",
    "batch_size = 5\n",
    "\n",
    "inputs = np.random.rand(batch_size, width, in_channels).astype(\n",
    "    np.float32)\n",
    "\n",
    "layer = layers.Conv1D(filters, kernel_size)\n",
    "\n",
    "result = layer(inputs)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again it should be noted that creating a second `Conv1D` layer would producr different results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So thats how we invoke different DeepChem layers in eager mode.\n",
    "\n",
    "One of the other interesting point is that we can mix tensorflow layers and DeepChem layers. Since they all take tensors as inputs and return tensors as outputs, so you can take the output from one kind of layer and pass it as input to a different kind of layer. But it should be noted that tensorflow layers can't be added to a TensorGraph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workflow of DeepChem layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've generalised so much, we should actually see if deepchem supplies an identical workflow for layers to that of tensorflow. For instance, let's consider the code where we create a `Dense` layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "y = Dense(3)(input)\n",
    "```\n",
    "\n",
    "What the above line does is that it creates a dense layer with three outputs. It initializes the weights and the biases. And then it multiplies the input tensor by the weights.\n",
    "\n",
    "Let's put the above statement in some mathematical terms. A `Dense` layer has a matrix of weights of shape `(M, N)`, where M is the number of outputs and N is the number of inputs. The first time we call it, the layer sets N based on the shape of the input we passed to it and creates the weight matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[-1.2747569  -1.2716365  -1.6619799 ]\n",
      " [ 0.01867434 -0.25217438  0.9760794 ]], shape=(2, 3), dtype=float32)\n",
      "tf.Tensor(\n",
      "[[-4.077428    1.9740283  -3.2914114   0.6148914 ]\n",
      " [ 1.3582385  -0.1323605   0.70408916 -1.2962227 ]], shape=(2, 4), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "_input = tf.random_normal([2, 3])\n",
    "print(_input)\n",
    "\n",
    "layer = layers.Dense(4) # A DeepChem Dense layer\n",
    "result = layer(_input)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is exactly how a tensorflow `Dense` layer works. It implements the same operation as that of DeepChem's `Dense` layer i.e., `outputs = activation(inputs.kernel + bias)` where `kernel` is the weights matrix created by the layer, and `bias` is a bias vector created by the layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[ 0.10799867 -3.0421743   1.1339507   0.85969573]\n",
      " [ 0.49897006  0.54293513 -0.617443   -0.551059  ]], shape=(2, 4), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "result = tf.layers.dense(_input, units=4) # A tensorflow Dense layer\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We pass a tensor input to that of tensorflow `Dense` layer and recieve an output tensor that has the same shape as that of input except the last dimension is that of ouput space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding gradients under eager mode is much similar to the `autograd` API. The computational flow is very clean and logical.\n",
    "What happens is that different operations can occur during each call, all forward operations are recorded to a tape, which is then played backwards when computing gradients. After the gradients have been computed, the tape is discared.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[[0.70184654]\n",
      "  [0.14101903]\n",
      "  [0.40004665]\n",
      "  [0.65922797]\n",
      "  [0.7357256 ]]\n",
      "\n",
      " [[0.2765366 ]\n",
      "  [0.07410333]\n",
      "  [0.6447927 ]\n",
      "  [0.6917898 ]\n",
      "  [0.29831788]]\n",
      "\n",
      " [[0.70695525]\n",
      "  [0.5157202 ]\n",
      "  [0.31305453]\n",
      "  [0.13231038]\n",
      "  [0.25589827]]\n",
      "\n",
      " [[0.6361409 ]\n",
      "  [0.4190178 ]\n",
      "  [0.43067047]\n",
      "  [0.17994796]\n",
      "  [0.49510527]]\n",
      "\n",
      " [[0.7120138 ]\n",
      "  [0.12389299]\n",
      "  [0.05375208]\n",
      "  [0.27753747]\n",
      "  [0.6016071 ]]], shape=(5, 5, 1), dtype=float32)\n",
      "[None]\n"
     ]
    }
   ],
   "source": [
    "def dense_squared(x):\n",
    "    return layers.Dense(1)(layers.Dense(1)(inputs))\n",
    "\n",
    "grad = tfe.gradients_function(dense_squared)\n",
    "\n",
    "print(dense_squared(3.0))\n",
    "print(grad(3.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example, The `gradients_function` call takes a Python function `dense_squared()` as an argument and returns a Python callable that computes the partial derivatives of `dense_squared()` with respect to its inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:deepchem]",
   "language": "python",
   "name": "conda-env-deepchem-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
